Understanding enterprise data warehouses to support clinical and translational research: impact, sustainability, demand management, and accessibility.

OBJECTIVES: Healthcare organizations, including Clinical and Translational Science Awards (CTSA) hubs funded by the National Institutes of Health, seek to enable secondary use of electronic health record (EHR) data through an enterprise data warehouse for research (EDW4R), but optimal approaches are unknown. In this qualitative study, our goal was to understand EDW4R impact, sustainability, demand management, and accessibility. MATERIALS AND METHODS: We engaged a convenience sample of informatics leaders from CTSA hubs (n = 21) for semi-structured interviews and completed a directed content analysis of interview transcripts. RESULTS: EDW4R have created institutional capacity for single- and multi-center studies, democratized access to EHR data for investigators from multiple disciplines, and enabled the learning health system. Bibliometrics have been challenging due to investigator non-compliance, but one hub's requirement to link all study protocols with funding records enabled quantifying an EDW4R's multi-million dollar impact. Sustainability of EDW4R has relied on multiple funding sources with a general shift away from the CTSA grant toward institutional and industry support. To address EDW4R demand, institutions have expanded staff, used different governance approaches, and provided investigator self-service tools. EDW4R accessibility can benefit from improved tools incorporating user-centered design, increased data literacy among scientists, expansion of informaticians in the workforce, and growth of team science. DISCUSSION: As investigator demand for EDW4R has increased, approaches to tracking impact, ensuring sustainability, and improving accessibility of EDW4R resources have varied. CONCLUSION: This study adds to understanding of how informatics leaders seek to support investigators using EDW4R across the CTSA consortium and potentially elsewhere.

Precision Phenotyping for Curating Research Cohorts of Patients with Post-Acute Sequelae of COVID-19 (PASC) as a Diagnosis of Exclusion.

Scalable identification of patients with the post-acute sequelae of COVID-19 (PASC) is challenging due to a lack of reproducible precision phenotyping algorithms and the suboptimal accuracy, demographic biases, and underestimation of the PASC diagnosis code (ICD-10 U09.9). In a retrospective case-control study, we developed a precision phenotyping algorithm for identifying research cohorts of PASC patients, defined as a diagnosis of exclusion. We used longitudinal electronic health records (EHR) data from over 295 thousand patients from 14 hospitals and 20 community health centers in Massachusetts. The algorithm employs an attention mechanism to exclude sequelae that prior conditions can explain. We performed independent chart reviews to tune and validate our precision phenotyping algorithm. Our PASC phenotyping algorithm improves precision and prevalence estimation and reduces bias in identifying Long COVID patients compared to the U09.9 diagnosis code. Our algorithm identified a PASC research cohort of over 24 thousand patients (compared to about 6 thousand when using the U09.9 diagnosis code), with a 79.9 percent precision (compared to 77.8 percent from the U09.9 diagnosis code). Our estimated prevalence of PASC was 22.8 percent, which is close to the national estimates for the region. We also provide an in-depth analysis outlining the clinical attributes, encompassing identified lingering effects by organ, comorbidity profiles, and temporal differences in the risk of PASC. The PASC phenotyping method presented in this study boasts superior precision, accurately gauges the prevalence of PASC without underestimating it, and exhibits less bias in pinpointing Long COVID patients. The PASC cohort derived from our algorithm will serve as a springboard for delving into Long COVID's genetic, metabolomic, and clinical intricacies, surmounting the constraints of recent PASC cohort studies, which were hampered by their limited size and available outcome data.

Erratum: Toward standardization, harmonization, and integration of social determinants of health data: A Texas Clinical and Translational Science Award institutions collaboration - CORRIGENDUM.

[This corrects the article DOI: 10.1017/cts.2024.2.].

Toward standardization, harmonization, and integration of social determinants of health data: A Texas Clinical and Translational Science Award institutions collaboration.

Introduction: The focus on social determinants of health (SDOH) and their impact on health outcomes is evident in U.S. federal actions by Centers for Medicare & Medicaid Services and Office of National Coordinator for Health Information Technology. The disproportionate impact of COVID-19 on minorities and communities of color heightened awareness of health inequities and the need for more robust SDOH data collection. Four Clinical and Translational Science Award (CTSA) hubs comprising the Texas Regional CTSA Consortium (TRCC) undertook an inventory to understand what contextual-level SDOH datasets are offered centrally and which individual-level SDOH are collected in structured fields in each electronic health record (EHR) system potentially for all patients. Methods: Hub teams identified American Community Survey (ACS) datasets available via their enterprise data warehouses for research. Each hub's EHR analyst team identified structured fields available in their EHR for SDOH using a collection instrument based on a 2021 PCORnet survey and conducted an SDOH field completion rate analysis. Results: One hub offered ACS datasets centrally. All hubs collected eleven SDOH elements in structured EHR fields. Two collected Homeless and Veteran statuses. Completeness at four hubs was 80%-98%: Ethnicity, Race; < 10%: Education, Financial Strain, Food Insecurity, Housing Security/Stability, Interpersonal Violence, Social Isolation, Stress, Transportation. Conclusion: Completeness levels for SDOH data in EHR at TRCC hubs varied and were low for most measures. Multiple system-level discussions may be necessary to increase standardized SDOH EHR-based data collection and harmonization to drive effective value-based care, health disparities research, translational interventions, and evidence-based policy.

Attitudes on Artificial Intelligence use in Pediatric Care From Parents of Hospitalized Children.

INTRODUCTION: Artificial intelligence (AI) may benefit pediatric healthcare, but it also raises ethical and pragmatic questions. Parental support is important for the advancement of AI in pediatric medicine. However, there is little literature describing parental attitudes toward AI in pediatric healthcare, and existing studies do not represent parents of hospitalized children well. METHODS: We administered the Attitudes toward Artificial Intelligence in Pediatric Healthcare, a validated survey, to parents of hospitalized children in a single tertiary children's hospital. Surveys were administered by trained study personnel (11/2/2021-5/1/2022). Demographic data were collected. An Attitudes toward Artificial Intelligence in Pediatric Healthcare score, assessing openness toward AI-assisted medicine, was calculated for seven areas of concern. Subgroup analyses were conducted using Mann-Whitney U tests to assess the effect of race, gender, education, insurance, length of stay, and intensive care unit (ICU) admission on AI use. RESULTS: We approached 90 parents and conducted 76 surveys for a response rate of 84%. Overall, parents were open to the use of AI in pediatric medicine. Social justice, convenience, privacy, and shared decision-making were important concerns. Parents of children admitted to an ICU expressed the most significantly different attitudes compared to parents of children not admitted to an ICU. CONCLUSIONS: Parents were overall supportive of AI-assisted healthcare decision-making. In particular, parents of children admitted to ICU have significantly different attitudes, and further study is needed to characterize these differences. Parents value transparency and disclosure pathways should be developed to support this expectation.

Predicting multiple sclerosis severity with multimodal deep neural networks.

Multiple Sclerosis (MS) is a chronic disease developed in the human brain and spinal cord, which can cause permanent damage or deterioration of the nerves. The severity of MS disease is monitored by the Expanded Disability Status Scale, composed of several functional sub-scores. Early and accurate classification of MS disease severity is critical for slowing down or preventing disease progression via applying early therapeutic intervention strategies. Recent advances in deep learning and the wide use of Electronic Health Records (EHR) create opportunities to apply data-driven and predictive modeling tools for this goal. Previous studies focusing on using single-modal machine learning and deep learning algorithms were limited in terms of prediction accuracy due to data insufficiency or model simplicity. In this paper, we proposed the idea of using patients' multimodal longitudinal and longitudinal EHR data to predict multiple sclerosis disease severity in the future. Our contribution has two main facets. First, we describe a pioneering effort to integrate structured EHR data, neuroimaging data and clinical notes to build a multi-modal deep learning framework to predict patient's MS severity. The proposed pipeline demonstrates up to 19% increase in terms of the area under the Area Under the Receiver Operating Characteristic curve (AUROC) compared to models using single-modal data. Second, the study also provides valuable insights regarding the amount useful signal embedded in each data modality with respect to MS disease prediction, which may improve data collection processes.

Technical/Algorithm, Stakeholder, and Society (TASS) barriers to the application of artificial intelligence in medicine: A systematic review.

INTRODUCTION: The use of artificial intelligence (AI), particularly machine learning and predictive analytics, has shown great promise in health care. Despite its strong potential, there has been limited use in health care settings. In this systematic review, we aim to determine the main barriers to successful implementation of AI in healthcare and discuss potential ways to overcome these challenges. METHODS: We conducted a literature search in PubMed (1/1/2001-1/1/2023). The search was restricted to publications in the English language, and human study subjects. We excluded articles that did not discuss AI, machine learning, predictive analytics, and barriers to the use of these techniques in health care. Using grounded theory methodology, we abstracted concepts to identify major barriers to AI use in medicine. RESULTS: We identified a total of 2,382 articles. After reviewing the 306 included papers, we developed 19 major themes, which we categorized into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). These themes included: Lack of Explainability, Need for Validation Protocols, Need for Standards for Interoperability, Need for Reporting Guidelines, Need for Standardization of Performance Metrics, Lack of Plan for Updating Algorithm, Job Loss, Skills Loss, Workflow Challenges, Loss of Patient Autonomy and Consent, Disturbing the Patient-Clinician Relationship, Lack of Trust in AI, Logistical Challenges, Lack of strategic plan, Lack of Cost-effectiveness Analysis and Proof of Efficacy, Privacy, Liability, Bias and Social Justice, and Education. CONCLUSION: We identified 19 major barriers to the use of AI in healthcare and categorized them into three levels: the Technical/Algorithm, Stakeholder, and Social levels (TASS). Future studies should expand on barriers in pediatric care and focus on developing clearly defined protocols to overcome these barriers.

Medication Reconciliation during Transitions of Care Across Institutions: A Quantitative Analysis of Challenges and Opportunities.

OBJECTIVE: Medication discrepancies between clinical systems may pose a patient safety hazard. In this paper, we identify challenges and quantify medication discrepancies across transitions of care. METHODS: We used structured clinical data and free-text hospital discharge summaries to compare active medications' lists at four time points: preadmission (outpatient), at-admission (inpatient), at-discharge (inpatient), and postdischarge (outpatient). Medication lists were normalized to RxNorm. RxNorm identifiers were further processed using the RxNav API to identify the ingredient. The specific drugs and ingredients from inpatient and outpatient medication lists were compared. RESULTS: Using RxNorm drugs, the median percentage intersection when comparing active medication lists within the same electronic health record system ranged between 94.1 and 100% indicating substantial overlap. Similarly, when using RxNorm ingredients the median percentage intersection was 94.1 to 100%. In contrast, the median percentage intersection when comparing active medication lists across EHR systems was significantly lower (RxNorm drugs: 6.1-7.1%; RxNorm ingredients: 29.4-35.0%) indicating that the active medication lists were significantly less similar (p < 0.05).Medication lists in the same EHR system are more similar to each other (fewer discrepancies) than medication lists in different EHR systems when comparing specific RxNorm drug and the more general RxNorm ingredients at transitions of care. Transitions of care that require interoperability between two EHR systems are associated with more discrepancies than transitions where medication changes are expected (e.g., at-admission vs. at-discharge). Challenges included lack of access to structured, standardized medication data across systems, and difficulty distinguishing medications from orderable supplies such as lancets and diabetic test strips. CONCLUSION: Despite the challenges to medication normalization, there are opportunities to identify and assist with medication reconciliation across transitions of care between institutions.

Confidence-based laboratory test reduction recommendation algorithm.

BACKGROUND: We propose a new deep learning model to identify unnecessary hemoglobin (Hgb) tests for patients admitted to the hospital, which can help reduce health risks and healthcare costs. METHODS: We collected internal patient data from a teaching hospital in Houston and external patient data from the MIMIC III database. The study used a conservative definition of unnecessary laboratory tests, which was defined as stable (i.e., stability) and below the lower normal bound (i.e., normality). Considering that machine learning models may yield less reliable results when trained on noisy inputs containing low-quality information, we estimated prediction confidence to assess the reliability of predicted outcomes. We adopted a "select and predict" design philosophy to maximize prediction performance by selectively considering samples with high prediction confidence for recommendations. Our model accommodated irregularly sampled observational data to make full use of variable correlations (i.e., with other laboratory test values) and temporal dependencies (i.e., previous laboratory tests performed within the same encounter) in selecting candidates for training and prediction. RESULTS: The proposed model demonstrated remarkable Hgb prediction performance, achieving a normality AUC of 95.89% and a Hgb stability AUC of 95.94%, while recommending a reduction of 9.91% of Hgb tests that were deemed unnecessary. Additionally, the model could generalize well to external patients admitted to another hospital. CONCLUSIONS: This study introduces a novel deep learning model with the potential to significantly reduce healthcare costs and improve patient outcomes by identifying unnecessary laboratory tests for hospitalized patients.

Why is biomedical informatics hard? A fundamental framework.

Building on previous work to define the scientific discipline of biomedical informatics, we present a framework that categorizes fundamental challenges into groups based on data, information, and knowledge, along with the transitions between these levels. We define each level and argue that the framework provides a basis for separating informatics problems from non-informatics problems, identifying fundamental challenges in biomedical informatics, and provides guidance regarding the search for general, reusable solutions to informatics problems. We distinguish between processing data (symbols) and processing meaning. Computational systems, that are the basis for modern information technology (IT), process data. In contrast, many important challenges in biomedicine, such as providing clinical decision support, require processing meaning, not data. Biomedical informatics is hard because of the fundamental mismatch between many biomedical problems and the capabilities of current technology.

Enoxaparin may be associated with lower rates of mortality than unfractionated heparin in neurocritical and surgical patients.

Unfractionated heparin (UFH) and low molecular weight heparin (LMWH) are often administered to prevent venous thromboembolism (VTE) in critically ill patients. However, the preferred prophylactic agent (UFH or LMWH) is not known. We compared the all-cause mortality rate in patients receiving UFH to LMWH for VTE prophylaxis. We conducted a retrospective propensity score adjusted analysis of patients admitted to neuro-critical, surgical, or medical intensive care units. Patients were included if they were screened with venous duplex ultrasonography or computed tomography angiography for detection of VTE. The primary outcome was all-cause mortality. Secondary outcomes included the prevalence of VTE, deep vein thrombosis (DVT), pulmonary embolism (PE), and hospital length of stay (LOS). Initially 2228 patients in the cohort were included for analysis, 1836 (82%) patients received UFH, and 392 (18%) patients received enoxaparin. After propensity score matching, a well-balanced cohort of 618 patients remained in the study (309 patients receiving UFH; 309 patients receiving enoxaparin). The use of UFH for VTE prophylaxis in ICU patients was associated with similar rates of all-cause mortality compared with enoxaparin [RR 0.73; 95% CI 0.43-1.24, p = 0.310]. There were no differences in the prevalence of DVT, prevalence of PE or hospital LOS between the two groups, DVT [RR 0.93; 95% CI 0.56-1.53, p = 0.889], PE [RR 1.50; 95% CI 0.78-2.90, p = 0.296] and LOS [9 ± 9 days vs 9 ± 8; p = 0.857]. A trend toward mortality benefit was observed in NICU [RR 0.37; 95% CI 0.13-1.07, p = 0.062] and surgical patients [RR 0.43; 95% CI 0.17-1.02, p = 0.075] favoring the enoxaparin group. The use of UFH for VTE prophylaxis in ICU patients was associated with similar rates of VTE, all-cause mortality and LOS compared to enoxaparin. In subgroup analysis, neuro-critical and surgical patients who received UFH had a higher rate of mortality than those who received enoxaparin.

A deep learning approach to identify missing is-a relations in SNOMED CT.

OBJECTIVE: SNOMED CT is the largest clinical terminology worldwide. Quality assurance of SNOMED CT is of utmost importance to ensure that it provides accurate domain knowledge to various SNOMED CT-based applications. In this work, we introduce a deep learning-based approach to uncover missing is-a relations in SNOMED CT. MATERIALS AND METHODS: Our focus is to identify missing is-a relations between concept-pairs exhibiting a containment pattern (ie, the set of words of one concept being a proper subset of that of the other concept). We use hierarchically related containment concept-pairs as positive instances and hierarchically unrelated containment concept-pairs as negative instances to train a model predicting whether an is-a relation exists between 2 concepts with containment pattern. The model is a binary classifier leveraging concept name features, hierarchical features, enriched lexical attribute features, and logical definition features. We introduce a cross-validation inspired approach to identify missing is-a relations among all hierarchically unrelated containment concept-pairs. RESULTS: We trained and applied our model on the Clinical finding subhierarchy of SNOMED CT (September 2019 US edition). Our model (based on the validation sets) achieved a precision of 0.8164, recall of 0.8397, and F1 score of 0.8279. Applying the model to predict actual missing is-a relations, we obtained a total of 1661 potential candidates. Domain experts performed evaluation on randomly selected 230 samples and verified that 192 (83.48%) are valid. CONCLUSIONS: The results showed that our deep learning approach is effective in uncovering missing is-a relations between containment concept-pairs in SNOMED CT.

Association of Frailty and the Expanded Operative Stress Score with Preoperative Acute Serious Conditions, Complications, and Mortality in Males Compared to Females: A Retrospective Observational Study.

OBJECTIVE: The aim of this study was to expand Operative Stress Score (OSS) increasing procedural coverage and assessing OSS and frailty association with Preoperative Acute Serious Conditions (PASC), complications and mortality in females versus males. SUMMARY BACKGROUND DATA: Veterans Affairs male-dominated study showed high mortality in frail veterans even after very low stress surgeries (OSS1). METHODS: Retrospective cohort using NSQIP data (2013-2019) merged with 180-day postoperative mortality from multiple hospitals to evaluate PASC, 30-day complications and 30-, 90-, and 180-day mortality. RESULTS: OSS expansion resulted in 98.2% case coverage versus 87.0% using the original. Of 82,269 patients (43.8% male), 7.9% were frail/very frail. Males had higher odds of PASC [adjusted odds ratio (aOR) = 1.31, 95% confidence interval (CI) = 1.21-1.41, P < 0.001] and severe/life-threatening Clavien-Dindo IV (CDIV) complications (aOR = 1.18, 95% CI = 1.09-1.28, P < 0.001). Although mortality rates were higher (all time-points, P < 0.001) in males versus females, mortality was similar after adjusting for frailty, OSS, and case status primarily due to increased male frailty scores. Additional adjustments for PASC and CDIV resulted in a lower odds of mortality in males (30-day, aOR = 0.81, 95% CI = 0.71-0.92, P = 0.002) that was most pronounced for males with PASC compared to females with PASC (30-day, aOR = 0.75, 95% CI = 0.56-0.99, P = 0.04). CONCLUSIONS: Similar to the male-dominated Veteran population, private sector, frail patients have high likelihood of postoperative mortality, even after low-stress surgeries. Preoperative frailty screening should be performed regardless of magnitude of the procedure. Despite males experiencing higher adjusted odds of PASC and CDIV complications, females with PASC had higher odds of mortality compared to males, suggesting differences in the aggressiveness of care provided to men and women.

Evolving availability and standardization of patient attributes for matching.

Variation in availability, format, and standardization of patient attributes across health care organizations impacts patient-matching performance. We report on the changing nature of patient-matching features available from 2010-2020 across diverse care settings. We asked 38 health care provider organizations about their current patient attribute data-collection practices. All sites collected name, date of birth (DOB), address, and phone number. Name, DOB, current address, social security number (SSN), sex, and phone number were most commonly used for cross-provider patient matching. Electronic health record queries for a subset of 20 participating sites revealed that DOB, first name, last name, city, and postal codes were highly available (>90%) across health care organizations and time. SSN declined slightly in the last years of the study period. Birth sex, gender identity, language, country full name, country abbreviation, health insurance number, ethnicity, cell phone number, email address, and weight increased over 50% from 2010 to 2020. Understanding the wide variation in available patient attributes across care settings in the United States can guide selection and standardization efforts for improved patient matching in the United States.

Patient and Provider Perspectives on Medication Non-adherence Among Patients with Depression and/or Diabetes in Diverse Community Settings - A Qualitative Analysis.

Background: Diabetes and depression affect a significant percentage of the world's total population, and the management of these conditions is critical for reducing the global burden of disease. Medication adherence is crucial for improving diabetes and depression outcomes, and research is needed to elucidate barriers to medication adherence, including the intentionality of non-adherence, to intervene effectively. The purpose of this study was to explore the perspectives of patients and health care providers on intentional and unintentional medication adherence among patients with depression and diabetes through a series of focus groups conducted across clinical settings in a large urban area. Methods: This qualitative study utilized a grounded theory approach to thematically analyze qualitative data using the framework method. Four focus groups in total were conducted, two with patients and two with providers, over a one-year period using a semi-structured facilitation instrument containing open-ended questions about experiences, perceptions and beliefs about medication adherence. Results: Across the focus groups, communication difficulties between patients and providers resulting in medication non-adherence was a primary theme that emerged. Concerns about medication side effects and beliefs about medication effectiveness were identified as perceptual barriers related to intentional medication non-adherence. Practical barriers to medication adherence, including medication costs, forgetting to take medications and polypharmacy, emerged as themes related to unintentional medication non-adherence. Conclusion: The study findings contribute to a growing body of research suggesting health system changes are needed to improve provider education and implement multicomponent interventions to improve medication adherence among patients with depression and/or diabetes, both chronic illnesses accounting for significant disease burden globally.

Real-World Matching Performance of Deidentified Record-Linking Tokens.

OBJECTIVE: Our objective was to evaluate tokens commonly used by clinical research consortia to aggregate clinical data across institutions. METHODS: This study compares tokens alone and token-based matching algorithms against manual annotation for 20,002 record pairs extracted from the University of Texas Houston's clinical data warehouse (CDW) in terms of entity resolution. RESULTS: The highest precision achieved was 99.9% with a token derived from the first name, last name, gender, and date-of-birth. The highest recall achieved was 95.5% with an algorithm involving tokens that reflected combinations of first name, last name, gender, date-of-birth, and social security number. DISCUSSION: To protect the privacy of patient data, information must be removed from a health care dataset to obscure the identity of individuals from which that data were derived. However, once identifying information is removed, records can no longer be linked to the same entity to enable analyses. Tokens are a mechanism to convert patient identifying information into Health Insurance Portability and Accountability Act-compliant deidentified elements that can be used to link clinical records, while preserving patient privacy. CONCLUSION: Depending on the availability and accuracy of the underlying data, tokens are able to resolve and link entities at a high level of precision and recall for real-world data derived from a CDW.

Understanding enterprise data warehouses to support clinical and translational research: enterprise information technology relationships, data governance, workforce, and cloud computing.

OBJECTIVE: Among National Institutes of Health Clinical and Translational Science Award (CTSA) hubs, effective approaches for enterprise data warehouses for research (EDW4R) development, maintenance, and sustainability remain unclear. The goal of this qualitative study was to understand CTSA EDW4R operations within the broader contexts of academic medical centers and technology. MATERIALS AND METHODS: We performed a directed content analysis of transcripts generated from semistructured interviews with informatics leaders from 20 CTSA hubs. RESULTS: Respondents referred to services provided by health system, university, and medical school information technology (IT) organizations as "enterprise information technology (IT)." Seventy-five percent of respondents stated that the team providing EDW4R service at their hub was separate from enterprise IT; strong relationships between EDW4R teams and enterprise IT were critical for success. Managing challenges of EDW4R staffing was made easier by executive leadership support. Data governance appeared to be a work in progress, as most hubs reported complex and incomplete processes, especially for commercial data sharing. Although nearly all hubs (n = 16) described use of cloud computing for specific projects, only 2 hubs reported using a cloud-based EDW4R. Respondents described EDW4R cloud migration facilitators, barriers, and opportunities. DISCUSSION: Descriptions of approaches to how EDW4R teams at CTSA hubs work with enterprise IT organizations, manage workforces, make decisions about data, and approach cloud computing provide insights for institutions seeking to leverage patient data for research. CONCLUSION: Identification of EDW4R best practices is challenging, and this study helps identify a breadth of viable options for CTSA hubs to consider when implementing EDW4R services.

Closing the loop: automatically identifying abnormal imaging results in scanned documents.

OBJECTIVES: Scanned documents (SDs), while common in electronic health records and potentially rich in clinically relevant information, rarely fit well with clinician workflow. Here, we identify scanned imaging reports requiring follow-up with high recall and practically useful precision. MATERIALS AND METHODS: We focused on identifying imaging findings for 3 common causes of malpractice claims: (1) potentially malignant breast (mammography) and (2) lung (chest computed tomography [CT]) lesions and (3) long-bone fracture (X-ray) reports. We train our ClinicalBERT-based pipeline on existing typed/dictated reports classified manually or using ICD-10 codes, evaluate using a test set of manually classified SDs, and compare against string-matching (baseline approach). RESULTS: A total of 393 mammograms, 305 chest CT, and 683 bone X-ray reports were manually reviewed. The string-matching approach had an F1 of 0.667. For mammograms, chest CTs, and bone X-rays, respectively: models trained on manually classified training data and optimized for F1 reached an F1 of 0.900, 0.905, and 0.817, while separate models optimized for recall achieved a recall of 1.000 with precisions of 0.727, 0.518, and 0.275. Models trained on ICD-10-labelled data and optimized for F1 achieved F1 scores of 0.647, 0.830, and 0.643, while those optimized for recall achieved a recall of 1.0 with precisions of 0.407, 0.683, and 0.358. DISCUSSION: Our pipeline can identify abnormal reports with potentially useful performance and so decrease the manual effort required to screen for abnormal findings that require follow-up. CONCLUSION: It is possible to automatically identify clinically significant abnormalities in SDs with high recall and practically useful precision in a generalizable and minimally laborious way.

External Validation of a Laboratory Prediction Algorithm for the Reduction of Unnecessary Labs in the Critical Care Setting.

BACKGROUND: Unnecessary laboratory tests contribute to iatrogenic harm and are a major source of waste in the health care system. We previously developed a machine learning algorithm to help clinicians identify unnecessary laboratory tests, but it has not been externally validated. In this study, we externally validate our machine learning algorithm. METHODS: To externally validate the machine learning algorithm that was originally trained on the Medical Information Mart for Intensive Care (MIMIC) III database, we tested the algorithm in a separate institution. We identified and abstracted data for all patients older than 18 years admitted to the intensive care unit at Memorial Hermann Hospital in Houston, Texas (MHH) from January 1, 2020 to November 13, 2020. Using the transfer learning style, we performed external validation of the machine learning algorithm. RESULTS: A total of 651 MHH patients were included. The model performed well in predicting abnormality (area under the curve [AUC] 0.98 for MIMIC III and 0.89 for MHH). The model performed similarly in predicting transitions from normal laboratory range to abnormal (AUC 0.71 for MIMIC III and 0.70 for MHH). The performance of the model in predicting the actual laboratory value was also similar in the MIMIC III (accuracy 0.41) and MHH data (0.45). CONCLUSIONS: We externally validated the machine learning model and showed that the model performed similarly, supporting the generalizability to other settings. While this model demonstrated good performance for predicting abnormal labs and transitions, it does not perform well enough for prediction of laboratory values in most clinical applications.